Skip to content

test: add colocated core component acceptance suite - #841

Open
kvinwang wants to merge 218 commits into
nextfrom
codex/core-components-test-infrastructure
Open

test: add colocated core component acceptance suite#841
kvinwang wants to merge 218 commits into
nextfrom
codex/core-components-test-infrastructure

Conversation

@kvinwang

@kvinwang kvinwang commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the core-component acceptance suite and organizes the complete suite directly under test-suites/ so specifications, automation, fixtures, inventories, manifests, and retained audit evidence have one clear home.

  • Adds the deterministic dstack-test runner, dashboard, registry validation, fixture providers, and component harnesses.
  • Adds 358 acceptance cases covering guest OS, VMM, KMS, gateway, verifier, and cross-component integration behavior.
  • Colocates single-case entrypoints with their case.md specifications as run.py, run.sh, or run.cjs.
  • Keeps multi-case and location-sensitive harnesses in shared/automation/.
  • Distributes metadata across suite, chapter, section, and case directories; the runner discovers cases in directory-name order.
  • Uses each case-local metadata.json as the authoritative case-to-entrypoint binding.
  • Keeps fixture providers and images under shared/fixtures/, inventories under catalog/, environment manifests under manifests/, and retained historical evidence under audit/.
  • Adds suite authoring, execution, reporting, and methodology documentation.
  • Excludes product/runtime changes and source-local unit tests; those changes were split into dedicated PRs.

Layout

test-suites/
├── runner/
├── cases/
├── shared/
│   ├── automation/
│   └── fixtures/
├── catalog/
├── manifests/
└── audit/

Separation invariant

The PR targets next. Its changed paths are limited to:

  • REUSE.toml
  • docs/testing/**
  • test-suites/**

There are no product/runtime or source-local test changes under dstack/** or os/**.

Verification

  • Registry: 357 scripted / 357 promoted / 0 findings (358 total cases)
  • Runner unit tests: 37 passed
  • prek run --all-files: passed
  • reuse lint: passed
  • git diff --check origin/next...HEAD: passed

Rebase and tdxlab validation

  • Rebases the branch onto next at afbafb90aa.
  • Final PR head: d8cbd45076.
  • Added and corrected regression coverage for changes merged since the previous PR baseline, including Guest RPC compatibility, Gateway KV and readiness behavior, GPU evidence, KMS trust, declarative OVMF selection, and structured RPC errors.
  • Built exact-revision production and development images with mkosi on tdxlab; the runtime manifest records d8cbd45076bea6d5778696b1c680b8ed8a7fe395.
  • Full physical-TDX sweep evidence: 341 PASS, 7 BLOCKED for unavailable NVIDIA confidential-computing GPU prerequisites, and 8 concurrency-sensitive failures. Every one of the 8 failures passed a one-worker rerun.
  • Final exact-revision validation passed for physical-TDX preflight, KMS provenance, KMS upgrade, both corrected failure-sequence cases, and the Certbot lifecycle.
  • Local gates passed: 40 runner unit tests, registry verification (357 scripted / 357 promoted / 358 total / 0 findings), reuse lint, and git diff --check.

The one product defect found during validation was fixed separately in #1139 and merged into next before the final rebase.

Copilot AI review requested due to automatic review settings July 31, 2026 02:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@kvinwang kvinwang changed the title test: add core component acceptance infrastructure [STACKED on #840] test: add core component acceptance infrastructure Jul 31, 2026
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from 7a21a97 to 313efb9 Compare July 31, 2026 03:34
@kvinwang kvinwang changed the title [STACKED on #840] test: add core component acceptance infrastructure test: add core component acceptance infrastructure Jul 31, 2026
@kvinwang
kvinwang changed the base branch from codex/core-components-product-fixes to master July 31, 2026 03:34
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from d7b3b76 to d8cbd45 Compare August 26, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants